Integrated camera image signal processor and video encoder

ABSTRACT

An apparatus including a first circuit and a second circuit. The first circuit may be configured to perform image signal processing using encoding related information. The second circuit may be configured to encode image data using image signal processing related information, wherein said first circuit is further configured to pass said image signal processing related information to said second circuit and said second circuit is further configured to pass said encoding related information to said first circuit.

This application is a continuation of and claims priority to U.S. patentapplication Ser. No. 11/410,729, entitled “Integrated Camera ImageSignal Processor and Video Encoder” filed on Apr. 25, 2006, which claimsthe benefit of U.S. Provisional Application No. 60/700,789, filed Jul.19, 2005 and both of which are hereby incorporated by reference in theirentirety.

FIELD OF THE INVENTION

The present invention relates to video processing generally and, moreparticularly, to a method and/or architecture for an integrated cameraimage signal processor and video encoder.

BACKGROUND OF THE INVENTION

Economies of scale in non-volatile storage and electronics in generalare making it possible to create commercially attractive devices capableof high quality digital still picture photography and high qualitydigital video recording in the same mobile device. However, typicaldevices are either efficient at taking digital still pictures (DigitalStill Cameras, DSC) or digital video streams (Digital Video Cameras,DVC), but not both.

SUMMARY OF THE INVENTION

The present invention concerns an apparatus including a first circuitand a second circuit. The first circuit may be configured to performimage signal processing using encoding related information. The secondcircuit may be configured to encode image data using image signalprocessing related information, wherein said first circuit is furtherconfigured to pass said image signal processing related information tosaid second circuit and said second circuit is further configured topass said encoding related information to said first circuit.

The objects, features and advantages of the present invention includeproviding an integrated camera image signal processor (ISP) and videoencoder that may (i) take advantage of processing and system synergies,(ii) provide an efficient digital still/video camera, (iii) provide alow cost digital still/video camera, (iv) provide improved performanceover separate camera implementations and/or (v) provide improved imagesignal processing and encoding/decoding by exchanging informationbetween image signal processor and CODEC functions.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other objects, features and advantages of the presentinvention will be apparent from the following detailed description andthe appended claims and drawings in which:

FIG. 1 is a block diagram illustrating an ISP/CODEC in accordance withthe present invention;

FIG. 2 is a more detailed block diagram illustrating connections betweenan image processing module and an encoding/decoding module of theISP/CODEC of FIG. 1;

FIG. 3 is a block diagram illustrating functions of the image signalprocessing module of FIG. 2;

FIG. 4 is a block diagram illustrating functions of the coder/decodermodule of FIG. 2;

FIGS. 5(A-B) are flow diagrams illustrating an example of a zoomoperation in accordance with a preferred embodiment of the presentinvention;

FIGS. 6(A-B) are flow diagrams illustrating an example of an autoexposure operation in accordance with a preferred embodiment of thepresent invention;

FIGS. 7(A-B) are flow diagrams illustrating an example of an auto focusoperation in accordance with a preferred embodiment of the presentinvention; and

FIGS. 8(A-B) are flow diagrams illustrating an example of a flashoperation in accordance with a preferred embodiment of the presentinvention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

A commercial mobile/portable system that is efficient as a digital stillcamera (DSC) and a digital video camera (DVC) involves the use ofspecific hardware/software. However, if the hardware/software systemsfor the DSC and the DVC are developed independently, allowing minimalinteraction with each other, processing and system synergies between thetwo systems are not fully utilized. The present invention generallyprovides a system for taking advantage of such synergies to produce anefficient and low cost combination DSC/DVC with better performance thanseparate implementations.

Referring to FIG. 1, a block diagram is shown illustrating a still andvideo picture processing system 100 in accordance with the presentinvention. In one example, the system 100 may include a block (orcircuit) 102, a block (or circuit) 104, a block (or circuit) 106, ablock (or circuit) 108, a block (or circuit) 110 and a block (orcircuit) 112. The block 102 may comprise an image signal (or sensor)processing (ISP) function (or circuit) and an encoding (orencoding/decoding (CODEC)) function (or circuit). In one example, theblock 102 may be implemented as a single integrated circuit chip.

The block 104 may be implemented, in one example, as a memory. However,other appropriate storage media or devices may be implementedaccordingly to meet the design criteria of a particular implementation.The block 106 may be implemented, in one example, as an image capturemodule. The block 108 may be implemented, in one example, as an imagedisplay module. The block 110 may be implemented, in one example, as adata transport module. The block 112 may be implemented, in one example,as a host controller.

In one example, the block 102 may have an input 114 that may receive asignal (e.g., IMAGE_DATA_IN), an input/output 116 that may be configuredto connect the block 102 with the block 104, an output 118 that maypresent a signal (e.g., BITSTREAM), an output 120 that may present asignal (e.g., PICTURE_DATA_OUT), an input 122 that may receive a signalor signals (e.g., CTRL_INFO_IN) and an output 124 that may present asignal or signals (e.g., CTRL_INFO_OUT). The signal BITSTREAM may beimplemented, in one example, as a compressed bitstream. In one example,the signal BITSTREAM may comprise data compressed in compliance with astandard or proprietary scheme (e.g., JPEG, MPEG, H.264, VC1, VC9,etc.). The signal IMAGE_DATA_IN may be implemented, in one example, as astill picture and/or video input signal. The signal PICTURE_DATA_OUT maybe implemented, in one example, as a still picture and/or video outputsignal. The signal(s) CTRL_INFO_IN may comprise a number of types ofinformation (e.g., capture control information, stabilizationparameters, flash settings, images, active parameters, exposuresettings, noise reduction settings, motion vectors and/or capture modeinformation). The signal(s) CTRL_INFO_OUT may comprise a number of typesof data (e.g., capture parameters, images, rate control information,statistical information, exposure settings, correlation analysis, etc.).

The block 104 may have an input/output 126 that may be configured tocouple (or interface) the block 104 to the block 102. The block 106 maybe configured to capture image data, either still or moving forpresentation to the block 102. For example, the block 106 may beimplemented, in one example, as a lens and sensor (e.g., CCD, etc.). Theblock 108 may be implemented in one example, as any of a number of typesof image display devices (e.g., CRT, LCD, plasma, etc.). The block 110may be implemented as any of a number of transport and/or storagesystems (e.g., CD, DVD, PVR, internet, satellite, broadcast, etc.). Theblock 112 may be configured to control various operations of the system100. Although interconnections to the other blocks are not shown, theblock 112 may be configured to provide control signals and receivecontrol data from each or all of the various blocks of the system 100,as would be recognized by one of ordinary skill in the art.

External images may be prepared for input to the block 102 by the block106. For example, the external images may be converted from an analogdomain to a digital domain. In one example, the images may be convertedby means of an image sensor and associated analog and digital processingcircuits. In one example, the images may be converted by the block 106from the analog domain into a format that is conducive to furtherdigital processing. For example, the block 106 may be configured toconvert and process data into a two-dimensional sampling matrix. In oneexample, the process used may include a generally known technique ofdemosaicing (or color interpolation). Conventional techniques may beused for demosaicing.

The blocks 108 and 110 are generally illustrative of output processes.The block 108 may be configured to display decoded bit streaminformation (e.g., on a monitor). The block 110 may be configured totransport the encoded bit stream (e.g., as an MPEG-2 transport stream,etc.). The block 112 is generally configured to oversee the entireprocess and coordinate control and data processing among all the variousmodules in the system 100.

Referring to FIG. 2, a more detailed block diagram is shown illustratingan example implementation of the block 102 of FIG. 1. In one example,the block 102 may comprise a block (or circuit) 130 and a block (orcircuit) 132. The block 130 may be implemented, in one example, as animage signal (or sensor) processing (ISP) module. In one example, theblock 130 may be implemented using a digital signal processor (DSP). Theblock 132 may be implemented, in one example, as a video/still pictureencoder/decoder (CODEC). In other examples, the block 132 may beimplemented as a video/still picture encoder, a video encoder or a stillpicture encoder. In one example, the block 104 may be implemented as amulti-port memory. Although the block 104 is generally illustrated as aseparate block from the block 102, one of skill in the relevant artwould recognize that the function of the block 104 may be integratedwith the block 102 and/or each of the blocks 130 and 132.

The block 130 may have a first input that may receive the signalIMAGE_DATA_IN, a first output that may present information (or data) asthe signal CTRL_INFO_OUT, a second input that may receive the signalCTRL_INFO_IN, a second output that may present a signal (e.g.,ISP_DATA), a third output that may present a signal (e.g.,PICTURE_DATA_IN), a third input that may receive a signal (e.g.,CODEC_DATA), and an input/output that may connect, in one example, to afirst port of the block 104. The block 132 may have a first input thatmay receive the signal PICTURE_DATA_IN, a second input that may receivethe signal ISP_DATA, a first output that may present information (ordata) as the signal CTRL_INFO_OUT, a third input that may receive thesignal CTRL_INFO_IN, a second output that may present the signalPICTURE_DATA_OUT, a third output that may present the signal BITSTREAM,a fourth output that may present the signal CODEC_DATA and aninput/output that may be coupled, in one example, to a second port ofthe block 104.

The block 130 may be configured to convert raw image data to picturedata amenable to compression. The block 130 may be configured to performseveral processing steps for creating a picture that is both true to theimage in the real world, and properly realized by the processingcircuits, taking into account limitations of the technology used for theprocess. In one example, the limitations may include (i) finite dynamicranges and temporal response in image sensor technology, (ii) inherentdistortions in lenses, (iii) inherent noise in the capture process, (iv)errors and inconsistencies in transducers used to control exposure,focus and color representation and/or (v) processing limitations incurrent signal processing techniques.

Properly exposed and corrected picture data may be presented to theblock 132 for further processing and compression. The block 132 may usehybrid entropy coding techniques such as MPEG, H.264, and/or JPEG toreduce the amount of picture data for storage and/or transmission. Inone example, decoded (or decompressed) data may be presented for displayon an external device. As used herein, the terms MPEG, H.264 and JPEGare used to generally refer to a variety of coding standards and/orproprietary schemes that may include, but are not limited to, MPEG-1,MPEG-2, MPEG-4, H.264, JPEG, JPEG-2000, etc.

Data (or image signal processing related information) generated fromprocesses in the block 130 may be transmitted via the signal ISP_DATA tothe block 132. Similarly, data (or coding/decoding related information)generated from processes in the block 132 may be transmitted via thesignal CODEC_DATA to the block 130. In one example, the data may betransmitted using a communication protocol that allows for control andinformation to be interchanged between the modules. The informationtransmitted from the block 130 to the block 132 and from the block 132to the block 130 is generally implemented for improving the quality ofprocessing in both blocks (or systems). For example, the processes ofthe block 130 and the block 132 may be configured to modify (or adapt,change, optimize, etc.) operations based upon the data received from theblock 132 and the block 130, respectively (described in more detailbelow in connection with FIGS. 5-8). The present invention may producestill pictures and video sequences of better quality than if the stillpictures or the sequences had been processed independently by separatemodules.

In general, processing improvements in the block 130 may be derived fromdata sent from the block 132. The improvements are generally independentof individual improvements in each of the processing steps involved in anormal image signal processor. Advancements in each processing operationin the block 130 may advance the performance of the system as a whole,but benefit may still be realized from data provided by the block 132 inaccordance with the present invention. The data sent from the block 132to the block 130 may comprise control data, statistical data and/orglobal and/or region-based spatio-temporal data derived from analysis ofthe image and video sequence data.

Similarly, video coding improvements by the block 132 may be realized byusing data received from the block 130 in accordance with the presentinvention. The improvements are generally independent of individualadvancements in each module in the block 132. The following sectionsgenerally describe examples of individual improvements that may beachieved by generating, transferring and using the data transferredbetween the block 130 and the block 132 in accordance with the presentinvention. The examples are presented as an illustration of, and not alimitation on, the improvements that may be realized using the presentinvention.

Referring to FIG. 3, a more detailed block diagram illustrating anexample of the image signal (or image sensor) processing module 130 ofFIG. 2 is shown. In one example, the block 130 may comprise a block (orcircuit) 140, a block (or circuit) 141, a block (or circuit) 142, ablock (or circuit) 143, a block (or circuit) 144, a block (or circuit)145, a block (or circuit) 146, a block (or circuit) 147, a block (orcircuit) 148, a block (or circuit 149, a block (or circuit) 150, a block(or circuit) 151, a block (or circuit) 152, a block (or circuit) 153 anda block (or circuit) 154. The block 140 may be implemented, in oneexample, as a geometric distortion correction block. The block 141 maybe implemented, in one example, as an auto exposure module. The block142 may be implemented, in one example, as a flash control module. Theblock 143 may be implemented, in one example, as a noise reductionmodule. The block 144 may be implemented, in one example, as a colorprocessor module. The block 145 may be implemented, in one example, as aauto focus module. The block 146 may be implemented, in one example, asa stabilization module. The block 147 may be implemented, in oneexample, as a red-eye reduction module. The block 148 may beimplemented, in one example, as an image analysis module. The block 149may be implemented, in one example, as a color interpolation block. Theblock 150 may be implemented, in one example, as an edge enhancementblock. The block 151 may be implemented, in one example, as a digitalzoom block. The block 152 may be implemented, in one example, as apicture scaling block. The block 153 may be implemented, in one example,as a resizing block. The block 154 may be implemented, in one example,as a control block.

The blocks 140-154 may be configured to perform a series of operationson the captured image data (e.g., received via the signal IMAGE_DATA_IN)in order to improve quality and compression of the data. In one example,the blocks 140-154 may be implemented by modifying conventionaltechniques in accordance with the teaching of the present disclosure.The blocks 140-154 may be implemented, in one example, using a digitalsignal processor (DSP). For example, a specialized processor may beimplemented containing dedicated hardware/software processing modules toachieve each operation illustrated in FIG. 3.

In general, real world images may be captured and converted from theanalog domain to the digital domain. The block 140 may provide forcorrecting geometric distortions in the optical system of the imagecapture block 106 (e.g., distortion and shading compensation). The block141 may provide automatic exposure functions to ensure that imageilluminance/color ranges are properly captured by the imaging sensor ofthe block 106. For low light situations, the block 143 may provide noisereduction techniques that may be implemented to improve thesignal-to-noise ratio (SNR) while the block 142 may provide automaticsettings of the external flash that may be used to improve sceneillumination. The block 144 may perform color processing (e.g., whitebalance, color correction, gamma correction, color conversion, etc.).The block 145 may be configured to automatically adjust a focal plane ofthe lens to ensure that the subject is in focus.

Under low lighting conditions with increased exposure time, thepossibility of image blur may be reduced by the block 146 implementingan electronic image stabilization process configured to operate inconcert with other processing modules in the block 130. Furthermore, inlow light environments, the use of an external flash may cause a‘red-eye’ condition in some subjects. The block 147 may implementautomatic red-eye reduction techniques that may be included in thesystem 100.

The image and picture quality provided by the block 130 may be furtherimproved through image analysis performed in the block 148, colorinterpolation processes performed by the block 149, edge enhancementprocesses implemented in the block 150 and/or digital zoom operationsperformed by the block 151. Image and picture size and/or aspect ratiomay be adjusted by the block 152 and the block 153. In general, theoperation of the blocks 140-153 may be controlled via the control block154. The block 130 is generally capable of producing quality images andpictures that may be further processed by a still picture compressionmodule and/or a video compression module. In one example, the stillpicture compression module and/or video compression module may beimplemented external to the block 130. For example, a still pictureencoder (or encoder/decoder) process and/or an advanced video encoder(AVC), or encoder/decoder (CODEC) process may reside in the block 132.

The signal ISP_DATA may be generated dynamically as the block 130 isoptimizing image capture for picture quality improvements. The signalCODEC_DATA sent from the block 132 to the block 130 may comprise controland statistics data and employ a mutually agreed upon transmissionprotocol that includes specific commands to adjust parameters of theblock 130.

As the pictures are encoded in real time, the block 130 may adjustoperations in order to improve performance using statistical and imagecharacteristics data transmitted back from the block 132 (described inmore detail below). For example, the block 130 may be configured tomodify processes based on data received from the block 132 in order toimprove picture quality. In general, all of the image signal processingtechniques implemented in the block 130 may benefit from real-time datagathered by the block 132 since image signal processing is generally notoptimized to process sequences of pictures but rather still images.

The control information received by the block 130 from the block 132 maybe used not only for improving image processing, but also to improvecoding efficiency. The ISP settings in the block 130 may be deliberatelymodified to affect both the still image and moving picture codingprocesses of the block 132. Furthermore, many of the techniquesdescribed herein may be tuned by a user of the system. For example, anoperator may set parameters to achieve a particular quality level instill picture/video capture situations.

The block 141 may be configured to provide automatic exposure processingfor the block 130. By examining the information from light sensors, theblock 141 may set exposure time and lens aperture to optimize imagecapture by the image sensor device. The sensitivity of the sensor iscommonly expressed in terms of traditional film sensitivity as dictatedby ISO. A properly exposed picture generally takes advantage of thedynamic range of the sensor. In some cases, the processing provided bythe block 130 may optimize the dynamic range so that particular areas ofthe picture are better exposed. For example, particular areas may bedeemed more important (e.g., people's faces in shadowed areas).

The block 142 may provide automatic flash control for the block 130.Under low lighting situations, the block 142 may automatically determinethe need for external flash illumination given the chosen (or derived)exposure parameters (e.g., aperture, shutter speed, ISO sensitivity).The block 142 may control the external flash via the signalCTRL_INFO_OUT.

The block 143 may provide noise reduction for the block 130. Under lowlighting conditions or high ISO equivalent capture settings, thepossibility of random noise generally increases. The block 143 mayreduce random noise to improve picture quality. In typical DSCconfigurations there is limited use of temporal domain information toreduce noise. Lowering the amount of random noise in pictures increasesperceived quality by allowing better compression. Lowering the amount ofrandom noise also allows better capture parameters to be used for eachpicture (e.g., higher shutter speeds and smaller apertures).

The block 145 may provide automatic focus control/processing for theblock 130. In one example, a normal operation of the automatic focuscontrol may be to hunt for a proper focus point. Such hunting behaviormay be modified based on information (e.g., motion vector, etc.)provided by the block 132 in accordance with the present invention. Forexample, a technique for focusing a camera may implement anauto-focusing mechanism that refocuses the image until a greatest amountof high frequencies are obtained. For example, the point where the mosthigh frequencies are obtained may be associated with the perfect focusof the point because the image no longer looks blurry. The presentinvention may use statistics about where a high number of edges arepresent in an image to control the auto-focusing mechanism. For example,the auto-focusing mechanism may be adjusted to focus on a small portionof the image containing edges to improve the performance of the autofocus.

In another example, the block 145 may analyze a set of areas of thepicture to determine the proper focal length for sharp image capture. Ingeneral, the block 130 may be configured to identify the main subject ofthe picture especially when the aperture is wide (i.e., thedepth-of-field is shallow) in order to properly bring into focus theimportant areas of the picture.

The block 146 may be configured to provide image stabilization. Underlow lighting situations or large zoom magnification, the possibility ofcamera motion (e.g., shake) and, therefore, blurry pictures increases.Large zoom factors generally employ larger apertures and thereforecreate shallow depth-of-field conditions that call for sharp, continuousfocusing. The block 154 generally manages the conflicting focusingdemands and modifies the efficacy of the stabilization techniques toimprove exposure and/or focus mechanisms.

The block 147 may be configured to perform red-eye reduction for theblock 130. Under low lighting conditions, some subjects will reflectflash illumination from the back of the eye (e.g., the retina) thereforecausing the red-eye effect. The block 147 may be configured to determinesuch situations and adapt/correct in the areas where red-eye is mostlikely to occur. In some implementations, the flash may be pulsedrapidly prior to final picture capture so as to narrow the pupilaperture of a subject.

The block 148 may be configured to derive picture characteristicsrepresenting spatial and temporal statistics data (e.g., averages,standard deviations, probability distributions, histograms, etc.). Theblock 148 may be further configured to collect feature-based imagecharacteristics (e.g., object texture, color, edges, motion, etc.) andintegrate the characteristics into regions of interest based on acombination of the picture features.

The majority of image sensors generate more spatial information thanallowed by the physical number of photosensitive receptors by samplingeach color component with different resolution. For example, asubsampling scheme referred to as Bayer pattern sampling may be used. Inthe Bayer pattern sampling scheme, the red and blue components aresampled once every four spatial positions, while the green component issampled twice every four spatial positions. The Bayer pattern samplingscheme uses a two-dimensional sampling pattern illustrated in thefollowing example array:

$\begin{matrix}G & B \\R & G\end{matrix}$The color interpolation block 149 may be used to generate a completesignal for all color components at all spatial positions through digitalinterpolation to produce the same number of samples for all the red,green and blue components.

The edge enhancement block 150 may be used to increase the perceivedresolution of sampled images. For example, the block 150 may detect thelocation and strength of object edges in two-dimensions and increase thecontrast of the edges with respect to the surrounding picture detail,therefore creating the appearance of more resolution.

The blocks 151, 152 and 153 are generally related to each other in thatthe blocks 151, 152 and 153 may change the resolution of the picturesbased upon predefined (or programmable) criteria. The digital zoom block151 may be used in place of an optical zoom mechanism. The optical zoomis performed at the lens and, although effective, is expensive torealize because of the cost of the lens. Instead, digital upscalingtechniques may be used to increase the resolution of an area ofinterest, therefore creating the same effect as the optical zoom. Thedigital zoom block 151 may emulate the analog lens operation of zoom-inand zoom-out at a rate of change determined by the user. In a digitalzoom operation the size of the picture remains the same.

The picture scaling block 152 may be implemented similarly to thedigital zoom block 151, except that the picture scaling block 152generally operates on the entire picture, while the digital zoomgenerally operates on an area of interest. Scaling may be used in caseswhere the size of the picture needs to be modified in the frame of theoriginal picture (e.g., to compose several pictures in one frame).Furthermore, picture scaling may be used to both increase and decreasethe resolution on the input images in scaling ratios determined by theapplication and not necessarily directly by the user.

A resizing operation may be implemented that is a special case ofpicture scaling. The total number of samples used to represent the imageare changed using digital resampling techniques (as opposed to the block151 and the block 152 which generally preserve the original frameresolution). The block 153 may implement the resizing operation in afaster more efficient manner than the blocks 151 and 152.

Referring to FIG. 4, a more detailed block diagram is shown illustratingan example of a video/still picture CODEC 132 implemented in accordancewith one embodiment of the present invention. In one example, the block132 may comprise a block (or circuit) 160, a block (or circuit) 162, ablock (or circuit) 164, a block (or circuit) 166 and a block (orcircuit) 168. The block 160 may be implemented in one example, as apre-processing module. The block 162 may be implemented, in one example,as an image analysis block. The block 164 may be implemented, in oneexample, as a still picture encoder/decoder (CODEC). The block 166 maybe implemented in one example, as a video encoder/decoder (CODEC). Inone example, the block 166 may be implemented as an H.264 compliantCODEC. However, the block 166 may be compliant with other standardsand/or proprietary compression schemes. The block 168 may beimplemented, in one example, as a control block.

In one example, the signal PICTURE_DATA_IN may be presented to an inputof the block 160. The block 160 may have a first output that may presenta signal to an input of the block 162 and a second output that maypresent a signal (e.g., PICTURE_DATA) to an input of the block 164 andan input of the block 166. The information sent to the block 162 mayinclude, but is not limited to, luminance picture data, chrominancepicture data, low frequency and high frequency information on luminanceand chrominance, object textures, object colors, object edges, temporaldifferences between pictures and motion estimated block vectors betweenpictures.

The block 162 may be configured to generate statistics and region data.The statistics and region data may be presented to a second input of theblock 164, a second input of the block 166 and a first input of theblock 168. The block 168 is implicitly connected to all of the blocks160-166 in the block 132. The block 168 may be implemented as atop-level controller that manages operating modes (dictated by a user),quality requirements, and performance requirements. For example, theblock 168 may be configured to manage rate control processes that maychange the behavior of the block 166. The block 168 sets parameters andissues instructions for coordination of operations of the other blocks.

The block 166 may have a first output that may present the signalPICTURE_DATA_OUT, a second output that may present the signal BITSTREAMand a third output that may present information via the signalCODEC_DATA. In general, data that is not ‘picture’, ‘image’,‘bitstream’, or any other image related data is processed by the systemcontrol block 168. The block 168 may have a second input that mayreceive the signal ISP_DATA, a third input that may receive the signalCTRL_INFO_IN, a first output that may present the signal CTRL_INFO_OUTand a second output that may present information via the signalCODEC_DATA. The block 168 may be configured to make decisions regardingsystem behavior. The block 168 generally executes the processesdescribed below in connection with FIGS. 5-8.

Image data proceeding from the block 130 is generally preprocessed bythe block 160 to improve compression quality (e.g., cropping, spatialscaling, color conversion and noise reduction). The image analysismodule 162 is generally configured to use original and preprocessed datato produce a series of global (picture-based) and regional (specificarea-based) statistics that represent image sequence characteristics.

The still picture encoder-decoder block 164 is generally configured tocode still pictures. The still picture coding process is always part ofa DSC since picture data compression is used to achieve efficientstorage and transmission. In general, the still picture coding process164 is included in the block 132 to illustrate that there is a unifiedcoding approach. The image signal processing analysis data received fromthe block 130 (e.g., via the signal ISP_DATA) may be used to improvestill picture quality. The process of using ISP analysis data to improvestill picture quality is included in the coding parameter optimizationphase illustrated in the flowcharts of FIGS. 5B, 6B and 7B.

In a conventional DSC, still picture coding is an external process tothe ISP and greatly influenced by the capture efficiency/processing ofthe ISP. Typically, still picture compression is achieved by JPEGcoding. By allowing both CODEC analysis data and ISP analysis data to befed into the still picture coding process, the present inventionprovides the ability to perceptually change the quantization parametersof selected areas of the picture.

The block 166 generally comprises several modules configured to processpicture sequences in space and time to produce high quality compressedvideo bit streams. In one example, the block 166 may comprise a block(or circuit) 180, a block (or circuit) 182, a block (or circuit) 184, ablock (or circuit) 186, a block (or circuit) 188, a block (or circuit)190, a block (or circuit) 192 and a block (or circuit) 194. The block180 may be implemented, in one example, as a motion estimation block.The block 182 may be implemented, in one example, as a transform andquantization block. The block 184 may be implemented, in one example, asa rate control block. The block 186 may be implemented, in one example,as a decoding block. The block 188 may be implemented, in one example,as a coding mode decision block. The block 190 may be implemented, inone example as a coding analysis block. The block 192 may beimplemented, in one example, as a deblocking filter block. The block 194may be implemented, in one example, as an entropy coder block. Theblocks 180-194 may be implemented using conventional circuits and/ortechniques modified in accordance with the teachings of the presentdisclosure. In one example, the block 186 may be omitted.

The blocks 180-194 are generally illustrative of processing modules in ahybrid entropy coding encoder. In general, the block 168 may beimplemented to coordinate the functioning of the individual modules180-194. Functioning of the encoder generally comprises control and datapath connections configured to achieve compliance with a particularcoding standard (or proprietary scheme).

In general, all of the modules 180-194 are influenced by the ratecontrol block 184 since the block 184 keeps track of bit rate andquality requirements for particular operating modes set by higher levelprocesses (e.g., user specifications and capture modes) controlled bythe block 168. The motion estimation block 180 and the coding modedecision block 188 generally determine the most efficient encoding forthe predicted data. The data may be predicted in the spatial direction(e.g., intra prediction) or the temporal direction (e.g., interprediction). The transform and quantization block 182 is generallyconfigured to convert the residual data (e.g., difference between actualand predicted data) into a domain most suitable for compression (e.g.,the process of discarding perceptually irrelevant data). Thequantization step is important for compression efficiency and one of themajor controls for bit rate and quality constrains managed by the ratecontrol block 184.

The decoding block 186 may be implemented as a standalone feature. Thedecoding block 186 generally duplicates the behavior of a decoder at theend of the transmission channel in order to avoid drift. Displayfunctions are generally associated with the block 186 since in manycases it is desirable to monitor the encoded bit stream.

The deblocking block 192 is generically placed in the diagram toindicate the inclusion of some filtering during the compression process,whether inside the coding loop or outside of the coding loop.Compression quality is generally improved by the use of deblockingcircuits.

The coding analysis block 190 may be configured to gather informationabout the coding process. The information gathered by the block 190 maybe sent to the block 130 via the signal CODEC_DATA to improveprocessing. The type of data transmitted to the block 130 is outlinedbelow.

In general, the block 132 may be configured to use data received fromthe block 130 via the signal ISP_DATA to change the behavior of themotion estimation (e.g., search window management, prediction modes,weighted prediction, etc.), quantization, rate control (e.g., to meetspecific quality vs. bit rate targets), and noise reduction modules. Inthe system depicted in FIG. 2, each of the blocks 130 and 132 is capableof generating analysis data relating to respective internal functioningand to the nature of the images processed. The image analysis module 156generates the data and sends it forward to the block 132 via the signalISP_DATA. The block 130 may be configured to generate data for use bythe block 132 as described below. For example, information on lensdistortion may be used to adjust coding parameters for optimalcompression in areas that have less optical fidelity. Quantization maybe coarser at the edges of the pictures where the distortion is morepronounced.

Hybrid entropy coding systems are partly based on detecting changes ofmotion in sequences. Therefore, there is always a need to estimatemotion between pictures. Typical motion estimation (ME) engines detectchanges based on some measure of correlation such as mean-squared error(MSE) or mean absolute error (MAE). These techniques tend to fail insituations where there is no real motion, but just changes in picturecharacteristics. Such cases may include sudden flashes of light (e.g.,when a camera attempts to illuminate a subject for proper exposure);fade in and fade out (e.g., when exposure changes in order to producethe optimum exposure). In such cases, block 130 provides informationabout the current state of exposure and focus settings to block 132 sothat motion estimation engines are set to operate under optimalconditions. For example, proper use of skip, direct and weightedprediction modes.

In state-of-the-art CODECs like H.264, there are inter-frame predictionmodes and intra-frame prediction modes. Selecting the appropriate modefor efficient coding is a process that involves multiple tries of intraand inter coding modes to find the best performance; or a-prioriknowledge of the material to select only the appropriate coding modes.Although the coding mode selection process is not strictly done in theME module, the coding mode selection process is included in this sectionbecause in one form or another selecting the correct intra or inter modeinvolves the analysis of displacement of picture data in either thespatial domain (intra modes) or the temporal domain (inter modes).

Quantization is the process of converting the transform coefficients tofixed point representation. As with any quantization process, smallcoefficients will be quantized to zero value if the quantizationparameter is large (coarse quantization parameter) and therefore, thosecoefficients can be regarded as non-critical for picture codingfidelity. However, there are many cases where particular picturefeatures are important for achieving good fidelity and therefore shouldnot be quantized coarsely, but rather with a finer (smaller)quantization step. Knowledge of image and sequence characteristics helpin determining those areas that should be quantized more carefully thanother to achieve good picture fidelity.

Achieving acceptable fidelity and high compression is a commonoptimization problem since fidelity and high compression constituteconflicting goals. In particular, excellent image quality may lead tolow (inefficient) compression, while high (efficient) compression tendsto produce poor image quality. The rate control block 184 is generallyconfigured to provide a good compromise between picture quality andcompression efficiency. A-priori knowledge of picture and sequencecharacteristics can help manage the optimization task described above.

Pictures devoid of random noise are easier to encode because there isless irrelevant high frequency information. However, the unabridgedapplication of noise reduction techniques to images in both the spatialand temporal domain can lead to soft and flat pictures that do notrepresent real scenery. Data regarding the capture process may beobtained in order to make better decisions as to the amount of noisereduction needed and the coding modes needed for faithful reproductionat all target bit rates.

The block 132 may be configured to send adjustments to exposureparameters of the block 130 via the signal CODEC_DATA based oncompression efficiency and quality performance. The exposure settingsmay be adjusted so that certain areas of the picture are optimallyencoded. The block 132 may identify overly dark and overly bright areasof the picture for better exposure and adjust the quantization approachso that those areas are assigned enough bits for proper coding. Thepicture gain of the block 130 may be adjusted via the signal CODEC_DATAto improve motion estimation performance and better rate control whenthe block 130 is in a dynamic configuration phase (e.g., auto exposure,auto focus, etc.). Fast changes in the picture gain due to auto exposure‘hunting’ may be minimized under very low bit rate conditions (e.g., soas not to overwhelm the encoding process).

The block 132 may be configured to send adjustments to the focusparameters of the block 130 via the signal CODEC_DATA based oncompression efficiency and quality performance. For example, focus maybe slightly de-tuned under extremely low bit rate requirements in orderto improve compression performance. Focus dynamics of the block 130 maybe adjusted during the auto focus phase in order to improve motionestimation performance and rate control (e.g., sudden focus changes maybe minimized in order to reduce sudden bit production during encoding).Adjustments to the noise reduction process of the block 130 may be sentvia the signal CODEC_DATA based on compression efficiency and qualityperformance. For example, under low lighting conditions and low motion,the block 132 may send commands for increased noise reduction sinceperformance will not be affected by high motion.

Global statistics data may be obtained by the preprocessing module 160after analyzing each picture as a whole (e.g., with no particular areaof each picture having more importance than another). Region-basedstatistics data may be obtained by classifying areas of the picture withcertain feature characteristics such as amount of low/high frequencytexture, amount of edges, amount of motion detected, amount of movingedges, type and amount of a specific color, etc.

In contrast with control data sent via the signal CODEC_DATA to theblock 130, the raw global and region-based statistics data may be sentas information to be processed and acted upon at the discretion of therespective image signal processing procedures of the block 130 (e.g., nodeclarative command actions are associated with the data). For example,size and strengths of regions in the picture classified according to aset of features (e.g., texture, edges, motion, color, etc.) may beprovided to the block 130 to evaluate the amount and relevance offeatured regions in the picture in both the spatial and temporaldomains. Information regarding areas of low/high texture in the picturemay be sent via the signal CODEC_DATA to the block 130. The block 130may use the texture information for auto-focus processing by evaluatingthe amount of low/high frequency in the picture as a result of changesof focal length by the focusing circuits.

Information regarding areas of the picture with strong edge componentsmay be sent from the block 132 to the block 130 via the signalCODEC_DATA. The block 130 uses the edge information for auto-focusingwhereby the amount of edges in the picture increase as the picture comesinto focus. This mechanism may be used along with motion information todetermine the quality of auto-focus data. Furthermore, edge informationcan be used in demosaicing procedures that rely on edge information toadapt the Bayer conversion process.

The block 132 may send information regarding areas of low/high motion inthe picture, including moving edges, to the block 130 via the signalCODEC_DATA. The block 130 may use the motion information forauto-focusing (e.g., to determine quiescent areas of the picture uponwhich to reliably focus), auto-exposure (e.g., by increasing exposuretime when pictures are non-moving) and noise reduction (e.g., byapplying stronger noise reduction to non-moving areas).

The block 132 may send further information concerning areas of thepicture featuring a specific color to the block 130 via the signalCODEC_DATA. In one example, the block 132 may be configured to operatein the YUV domain to facilitate detection of areas of specific colors.The color information may be used to identify human faces to help reducered-eye effects (which also depend upon identification of red areas ofthe picture). The block 130 may use the color information to reduce thered-eye effects when using a flash.

In addition to the control data and global picture and region-basedstatistics, the block 132 may be configured to send statistics derivedfrom analysis of the encoding process via the signal CODEC_DATA using amutually agreed protocol for data interchange. For example, motionvector information may be transmitted to the block 130 toimplement/improve image stabilization processes. The block 130 may usethe motion vector information to obtain detailed motion in specificareas of the picture for improved motion-adaptive procedures under lowlighting conditions (e.g., auto exposure, auto focus, noise reduction,image stabilization). In addition to the motion vector information,block distortion information may be sent to the block 130. For example,the block distortion information may include an indication of the codingfidelity at a given bit rate. Distortion per block at the edges of thepicture analyzed in the temporal domain may be an indication of fixedlens aberrations that may be corrected by the block 130.

In addition to sending the above information to the block 130 via thesignal CODEC_DATA, information generated in the block 132 may also besent to the block 164. The signal CODEC_DATA generally represents allinformation gathered by block 132 in the process of preprocessing andencoding the moving picture data. In one example, the block 164 may beimplemented as a JPEG or JPEG2000 still picture codec. The signalCODEC_DATA may contain information to improve performance of block 130as indicated on the left side of the flow charts in FIGS. 5-8. Forexample, in FIG. 8, when the flash is active, the block 130 may useinformation in the signal CODEC_DATA to change the flash intensity andduration.

In one example, the block 164 may be implemented as part of the block132. However, in other implementations the block 164 may be implementedas part of the block 130, or as a separate system altogether. In oneexample, the block 164 may be a separate system implemented with ageneral DSP Processor.

The block 132 may be configured to send the global picture andregion-based statistics data comprising texture information, edgeinformation, motion information and color information to the block 164.Texture information concerning areas of low/high texture in the picturemay be used by the block 164 to apply perceptual coding to those areas.For example, areas of high frequency texture may be coded according to atarget bit rate; while areas of low frequency texture may be coded witha fine quantization step in order to avoid banding artifacts. Edgeinformation regarding areas of the picture with strong edge componentsmay be used to improve perceptual coding. For example, areas with astrong edge component may be coded with a fine quantization step inorder to avoid edge artifacts like ringing and mosquito noise.

Motion information concerning areas of low/high motion in the picture,including moving edges may be used to improve perceptual coding. Forexample, areas of low motion may be coded with a fine quantization stepin order to reproduce them faithfully. On the other hand, areas of highmotion are presumably blurred and, depending on the ISP parametersreceived, may be coded with very few bits if over/under exposed oradversely affected by low shutter speeds. Color information about areasof the picture featuring a specific color may be used to identify humanfaces for perceptual coding. The quantization step may be much lower inthese areas in order to reproduce faces more faithfully.

In addition to the global picture and region-based statistics, the block132 may be configured to send statistics derived from analysis of theencoding process to the block 164 to steer perceptual processingperformed by the block 164. In one example, the statistics sent to theblock 164 may include a number of bits coded per block, blockdistortion, block coded residual variance and number of quantizedtransform coefficients with zero value per picture. The number of bitscoded per block may be sent to the block 164 to give an indication ofcoding efficiency per picture. The block 164 may collect the number ofbits coded per block information for every picture in order to improveinternal processing procedures and also to make decisions regarding theeffect of the internal processing procedures on compression efficiency.

The block distortion information may be sent to the block 164 as anindication of the coding fidelity at a given bit rate; together with thenumber of bits per block, the block distortion information may be usedto evaluate the total efficiency of the encoding process in terms of thecompromises between distortion and rate. For example, the block 132generally uses the number of bits per block and block distortion valuesto tune rate-control processes. The block coded residual variance dataare an indication of coding efficiency in both the spatial and temporaldomains as a result of coding decisions in the block 132 due to motionestimation and quantization. The block 164 may use the block codedresidual variance data to adapt internal processing methods byallocating less bits to areas where the residual is low (e.g., assumingthat the areas where the residual is low can be coded efficiently with ahigher quantization step). The decision to allocate less bits to areaswhere the residual is low may be used together with indications oftexture and motion in order to properly allocate bits to regions thatare more likely to produce artifacts at higher compression ratios. Thedata regarding the number of quantized transform coefficients with zerovalue per picture are an indication of compression efficiency withrespect to picture/sequence complexity. The information may be used bythe block 164 in the same manner as the number of bits per block.

Referring to FIGS. 5(A-B), flow diagrams are shown illustrating anexample of a zoom operation of the block 130 (FIG. 5A) and the block 132(FIG. 5B) of FIG. 2. In one example, a zoom procedure 200 for the block130 may comprise a state 201, a state 202, a state 204, a state 206 anda state 208. The zoom procedure 200 for the block 130 may start in thestate 201 and move to the state 202. The state 202 may comprise adecision state. In the state 202, the process 200 may determine whethera zoom function is active. When the zoom function is active, the process200 may move to the state 204. If the zoom function is not active, theprocess 200 may move to the state 208.

In the state 204, the process 200 may determine whether the zoomoperation is zooming in (e.g., enlarging or magnifying an object ofinterest). The operation of zooming in may also be referred to as thezoom moving in. When the zoom operation is not zooming in, the process200 may transition to the state 206. When the zoom operation is zoomingin, the process 200 may determine the speed (e.g., rate of change ofmagnification) at which the zoom is moving in and transition to thestate 208. In the state 206, the process 200 may determine whether thezoom operation is zooming out (e.g., reducing an apparent size orde-magnifying an object of interest). The operation of zooming out mayalso be referred to as the zoom moving out. When the zoom operation isnot zooming out, the process 200 may collect information regarding thecurrent magnification and transition to the state 208. When the zoomoperation is zooming out, the process 200 may determine a speed (e.g.,rate of change of the magnification) at which the zoom is moving out andtransition to the state 208. In the state 208, the information (e.g.,magnification and/or rate of change of the magnification) may becollected and sent to the block 132.

A zoom procedure 210 for the block 132 may comprise a state 212, a state214, a state 216, a state 218, a state 220, a state 222, a state 224 anda state 226. The state 212 may be implemented as a start state. Thestate 214 may be implemented as a zoom setting collection state. Thestate 216 may be implemented as a decision state. The state 218 may beimplemented as a decision state. The state 220 may be implemented as adecision state. The state 222 may be implemented as an adjustment state.The state 224 may be implemented as an optimization state. The state 226may be implemented as a completion state.

The process 210 may begin in the state 212 and transition to the state214. In the state 214, the process 210 generally obtains zoom settings.When the zoom setting have been obtained, the process 210 generallytransitions to the state 216. In the state 216, the process 210determines whether the zoom is active. When the zoom is not active, theprocess 210 generally transitions to the state 226 and terminates. Whenthe zoom is active, the process 210 generally transitions to the state218.

In the state 218, the process 210 determines whether the zoom operationis zooming in. When the zoom operation is not zooming in, the process210 generally transitions to the state 220. When the zoom operation iszooming in, the process 210 generally determines the speed (e.g., rateof change of magnification) at which the zoom is moving in andtransitions to the state 222. In the state 222, the process 210generally adjusts motion estimation and/or rate control parameters andtransitions to the state 226 for completion.

In the state 220, the process 210 generally determines whether the zoomoperation is zooming out. When the zoom operation is not zooming out,the process 210 generally determines the magnification and transitionsto the state 224. When the zoom operation is zooming out, the process210 generally determines the speed (e.g., rate of change ofmagnification) at which the zoom is moving out and transitions to thestate 222. In the state 224, the process 210 generally optimizes codingparameters based on the magnification determined and transitions to thestate 226.

The block 132 may be configured to modify motion estimation (ME) andmode decision processes specifically for zooming-in or zooming-outportions of the input sequence. Under fast zoom situations, the block132 may be configured to adjust rate control processes for efficientcoding. When zooming is in a quiescent state, picture parameters may beoptimized for a current magnification factor. For example, under highmagnification unstable camera hold may be a problem and therefore ratecontrol should be set to ignore quick changes. The process representedby the optimize coding parameters block 224 generally involvesnon-dynamic changes to the processes indicated in adjacent adjust motionestimation and adjust rate control blocks 222.

Referring to FIGS. 6(A-B), flow diagrams are shown illustrating anexample of an auto exposure process for the block 130 (FIG. 6A) and theblock 132 (FIG. 6B). The process 230 may comprise state 232, a decisionstate 234 and a state 236. In the state 232, the block 130 generallybegins the auto exposure procedure and transitions to the state 234. Inthe state 234, the process 230 determines whether or not the autoexposure mode is active. If the auto exposure mode is active, theprocess 230 transitions to the state 236 and collects informationregarding F-stop and shutter speed. When the auto exposure mode is notactive, the process 230 generally transitions to the state 236 and nodata is collected or sent. During the auto exposure procedure 240, theblock 132 generally enters the state 242 to begin the procedure for autoexposure. The process 240 generally transitions to the state 244. In thestate 244, the process 240 determines whether the automatic exposuremode is active. If the automatic exposure mode is not active, theprocess 240 generally transitions to the state 246. If the auto exposuremode is active, the process 244 generally transitions to the state 248.In the state 246, the process 240 generally optimizes coding parametersand transitions to the state 252. In the state 248, the process 240generally adjusts rate control parameters, adjusts quantizationparameters and adjusts motion estimation parameters. The adjustments tothe rate control, quantization and motion estimation may be performedtaking into account statistics received from the CODEC (e.g., the block152). Once the adjustments of rate control, quantization and motionestimation have been completed, the process 240 may transition to thestate 250 for completion.

When the block 130 adjusts exposure settings dynamically, the block 132may be configured to adjust quantization and rate control for efficientcoding. Change in global exposure settings may be handled with weightedprediction modes by the block 132 therefore improving coding rates.Dynamic range changes to selected areas of the picture may be coded withmore/less bits depending on the relevance of the regions in terms ofpsycho-visual measures (e.g., more important areas will be allocatedmore bits). Both data from the block 130 and statistics from the block132 may be used to improve the final coding efficiency of the output bitstreams. In general, the optimized coding parameters block 246 generallyrepresents non-dynamic changes to the processes indicated in theadjacent rate control, quantization and ME blocks 248.

Referring to FIGS. 7(A-B), flow diagrams are shown illustrating anexample of an auto focus process for the block 130 (FIG. 7A) and theblock 132 (FIG. 7B). The process 260 may comprise state 262, a decisionstate 264 and a state 266. In the state 262, the block 130 generallybegins the auto focus procedure and transitions to the state 264. In thestate 264, the process 260 determines whether or not the auto focus modeis active. When the auto focus mode is active, the process 260transitions to the state 266 and collects information regarding focusspeed. When the auto focus mode is not active, the process 260 generallytransitions to the state 266 and collects information regarding focallength.

During the auto focus procedure, the block 132 generally performs aprocess 270. The process 270 generally comprises a state 272, a state274, a state 276, a state 278 and a state 280. The process 270 generallyentered the state 272 to begin the procedure for auto focus. The process270 generally transitions to the state 274. In the state 274, theprocess 270 determines whether the automatic focus mode is active. Whenthe automatic focus mode is not active, the process 270 generallytransitions to the state 276. When the auto focus mode is active, theprocess 270 generally transitions to the state 278.

In the state 276, the process 270 generally optimizes coding parametersand transitions to the state 280. In one example, focal lengthinformation may be used to optimize the coding parameters. In the state278, the process 270 generally adjusts rate control parameters, adjustsquantization parameters and adjusts motion estimation parameters. Theadjustments to the rate control, quantization and motion estimation maybe performed taking into account speed information (e.g., rate of changeof focus). Once the adjustments of rate control, quantization and motionestimation have been completed, the process 270 may transition to thestate 280 for completion.

During focusing adjustments, the block 130 may be configured to changefocal length dynamically while the block 132 may be configured to adjustrate control for optimal quality (e.g., using lower bit rates to savebits for later more complex scenes). During the focusing operation theblock 132 generally uses low bit rates since the video is not yetoptimized for recording (blurry). The block 132 may be configured tomodify motion estimation processes to work more efficiently during focalchanges. The optimize coding parameters block 276 generally representsnon-dynamic changes to the processes indicated in the adjacent ME,Quantization and Rate Control blocks 278.

Referring to FIGS. 8(A-B), flow diagrams are shown illustrating anexample of flash process 300 for the block 130 (FIG. 8A) and a flashprocess 310 for the block 132 (FIG. 8B). The process 300 may comprisestate 302, a decision state 304 and a state 306. In the state 302, theblock 130 generally begins the flash procedure and transitions to thestate 304. In the state 304, the process 300 determines whether or notthe flash is active. When the flash is active, the process 300transitions to the state 306 and collects information regarding flashintensity and duration. When the flash is not active, the process 300generally transitions to the state 306 and no data is collected or sent.

The process 310 generally comprises a state 312, a state 314, a state316, and a state 318. During the flash procedure 310, the block 132generally enters the state 312 to begin the procedure for flashcompensation. The process 310 generally transitions to the state 314. Inthe state 314, the process 310 determines whether the flash is active.When the flash is active, the process 310 generally transitions to thestate 316. When the flash is not active, the process 310 generallytransitions to the state 318.

In the state 316, the process 310 generally adjusts rate controlparameters, adjusts quantization parameters and adjusts motionestimation parameters. The adjustments to the rate control, quantizationand motion estimation parameters may be performed taking into accountinformation about flash intensity an/or duration. Once the adjustmentsof the rate control, quantization and motion estimation parameters havebeen completed, the process 310 may transition to the state 318 forcompletion.

Sudden flash pictures can overwhelm a common video encoder. Bytransmitting flash intensity and duration information from the block 130to the block 132, the block 132 may adjust motion estimation and ratecontrol to deal with the sudden change in pictures. For example, at theflash transitions, the block 132 may insert new Intra prediction frames.In H.264 coding, a Long Term Picture may also be generated at the flashoccurrence.

As would be apparent to those skilled in the relevant art(s), thesignals illustrated in FIGS. 1-4 represent logical data flows. Thelogical data flows are generally representative of physical datatransferred between the respective blocks by, for example, address,data, and control signals and/or busses. The system 100 may beimplemented in hardware, software or a combination of hardware andsoftware according to the teachings of the present disclosure, as wouldbe apparent to those skilled in the relevant art(s).

The functions performed by the flow diagrams of FIGS. 5-8 may beimplemented using a conventional general purpose digital computerprogrammed according to the teachings of the present specification, aswill be apparent to those skilled in the relevant art(s). Appropriatesoftware coding can readily be prepared by skilled programmers based onthe teachings of the present disclosure, as will also be apparent tothose skilled in the relevant art(s).

The present invention may also be implemented by the preparation ofASICs, FPGAs, or by interconnecting an appropriate network ofconventional component circuits, as is described herein, modificationsof which will be readily apparent to those skilled in the art(s).

The present invention thus may also include a computer product which maybe a storage medium including instructions which can be used to programa computer to perform a process in accordance with the presentinvention. The storage medium can include, but is not limited to, anytype of disk including floppy disk, optical disk, CD-ROM,magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, Flash memory,magnetic or optical cards, or any type of media suitable for storingelectronic instructions.

While the invention has been particularly shown and described withreference to the preferred embodiments thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade without departing from the spirit and scope of the invention.

The invention claimed is:
 1. A camera image signal processor and videoencoder apparatus comprising: an image signal processor that modifies animage signal relative to a limitation of a camera that generated theimage signal, the limitation of the camera being associated with atleast one characteristic selected from finite dynamic ranges andtemporal response of the camera, inherent distortions in camera lenses,inherent noise in a camera capture process, and errors andinconsistencies in transducers used to control exposure, focus, andcolor representation; a video encoder, coupled directly to the imagesignal processor, to (i) receive the modified image signal and encodethe modified image signal using image signal processing relatedinformation, and (ii) transmit encoding related information to the imagesignal processor; and a bi-directional communication path coupledbetween the image signal processor and the video encoder that allowsinformation, including the image signal processing related information,to be transmitted between the image signal processor and the videoencoder, wherein said image signal processing related informationcomprises global picture and region-related statistics comprising one ormore types of data selected from the group consisting of size andstrengths of regions, texture information, edge information, motioninformation, color information and statistics derived from an analysisof an encoding process, wherein said statistics derived from analysis ofsaid encoding process comprise one or more types of data selected fromthe group consisting of motion vector information and block distortioninformation.
 2. The apparatus according to claim 1, wherein the imagesignal processor and the video encoder are integrated in a singleintegrated circuit.
 3. The apparatus according to claim 1, wherein videoencoder comprises one or more encoders selected from the groupconsisting of a video picture encoder, a still picture encoder, a videopicture encoder/decoder (CODEC) and a still picture encoder/decoder(CODEC).
 4. The apparatus according to claim 3, wherein said videopicture CODEC comprises an H.264 compliant CODEC.
 5. The apparatusaccording to claim 3, wherein said still picture CODEC comprises a JPEGcompliant CODEC.
 6. The apparatus according to claim 1, wherein theimage signal processor comprises an image signal processing (ISP)circuit.
 7. The apparatus according to claim 1, wherein the image signalprocessor comprises a digital signal processor (DSP).
 8. The apparatusaccording to claim 1, wherein the image signal processor and the videoencoder are further configured to (i) receive one or more first controlsignals from one or more external devices and (ii) present one or moresecond control signals to said one or more external devices.
 9. Theapparatus according to claim 1, further comprising: an image capturecircuit coupled to the image signal processor; and an image displaydevice coupled to the video encoder.
 10. The apparatus according toclaim 1, wherein the video encoder is further configured to present acompressed bitstream.
 11. An image signal processing and video encodingapparatus comprising: an image signal processing module, coupled toreceive an image signal from a camera, wherein the image signalprocessing module processes the image signal and generates a picturedata signal such that a characteristic of the image signal is modified,the first characteristic of the image signal is related to a limitationof the camera; a codec, coupled to the image signal processing module,to receive the picture data signal, the codec encodes the picture datasignal into an encoded signal, said codec further transmitting encodeddata and codec related information to the image signal processingmodule; wherein the limitation of the camera is associated with at leastone characteristic selected from finite dynamic ranges and temporalresponse of the camera, inherent distortions in camera lenses, inherentnoise in a camera capture process, and errors and inconsistencies intransducers used to control exposure, focus, and color representation,wherein said image signal processing module generates image signalprocessing related information comprising global picture andregion-related statistics comprising one or more types of data selectedfrom the group consisting of size and strengths of regions, textureinformation, edge information, motion information, color information andstatistics derived from an analysis of an encoding process, wherein saidstatistics derived from analysis of said encoding process comprise oneor more types of data selected from the group consisting of motionvector information and block distortion information.
 12. The apparatusof claim 11 wherein the camera is integrated within a mobile phone. 13.The apparatus of claim 11 wherein the camera is integrated within amobile handset.
 14. A method for processing and encoding image datacomprising the steps of: receiving an image signal from a camera on ahandset; modifying a characteristic of the image signal by a processor,related to a limitation of the camera, the limitation of the camerabeing associated with at least one characteristic selected from finitedynamic ranges and temporal response of the camera, inherent distortionsin camera lenses, inherent noise in a camera capture process, and errorsand inconsistencies in transducers used to control exposure, focus, andcolor representation; generating image signal processing relatedinformation based on the modified characteristic of the image signal;transmitting the modified image signal and the image signal processingrelated information to a video codec; and encoding the modified imagesignal pursuant to an encoding standard and using the image signalprocessing related information, wherein the video codec furthertransmits encoded data and codec related information to the processor,wherein the image signal processing related information comprises globalpicture and region-related statistics, and generating the image signalprocessing related information comprises selecting one or more types ofdata from a group consisting of size and strengths of regions, textureinformation, edge information, motion information, color information andstatistics derived from analysis of an encoding process, wherein saidstatistics derived from analysis of said encoding process comprise oneor more type of data selected from the group consisting of motion vectorinformation and block distortion information.
 15. The method accordingto claim 14, wherein generating the image signal processing relatedinformation comprises modifying commands that modify one or moreparameters selected from the group consisting of exposure parameters,gain parameters, focus parameters, flash parameters and noise reductionparameters.